@liruilong940607 I think it works quite well as of now but I haven't run the benchmarks yet.

What should we do about the file format? In theory, the mlp parameters need to be saved as well but that would create an own file format. Any thoughts?

I removed the pruning and use relocation. It seems to give better results.
Here is one of the eval videos at 30k steps. I need to run the benchmarks :)

Great job @MrNeRF! I think saving in it's own file format is fine -- it is what it is

Thank you for your suggestions @liruilong940607

I still play a little bit with the parameters and settings before I start cleaning it up. If the measure are not somewhat flawed (and I need to confirm on more data), this could be the new sota.

I get now on the garden scene:
PSNR: 27.801, SSIM: 0.8678, LPIPS: 0.076 Time: 0.010s/image Number of GS: 1021525

Update: The last commit gives me
PSNR: 27.941, SSIM: 0.8705, LPIPS: 0.074 Time: 0.012s/image Number of GS: 1213319

Hey this is insane. The default MCMC gives
PSNR: 26.983, SSIM: 0.8479, LPIPS: 0.113 Time: 0.059s/image Number of GS: 1000000

I will help for benchmarking. Training now. Hopefully it runs without hassle

Here are my results. I think I overfitted a bit on the garden scene. What is the base line? According to nerfbaselines it is sota (psnr) on the following scenes: garden, counter, bonsai. The rest is pretty close. Only on the bike scene it is average. However, I don't think the measures are comparable.

In terms of lpips it is everywhere sota (is this comparable at all). I think the visual appearance is in general also better.

I use the script under examples/benchmark. Why are the images downscaled for the training of certain scenes?

Eval Stats This pr

Here is the same benchmark with the mcmc implementation

@MrNeRF
Its strange, stump is not error with my experiment and using this commit e00d134

=== Eval Stats of stump ===
results/benchmark_scaffold/stump/stats/val_step29999.json
{"psnr": 26.83869171142578, "ssim": 0.7819640040397644, "lpips": 0.1407296359539032, "ellipse_time": 0.014932617545127869, "num_GS": 1560666}

Here is the latest state of this draft pr

@ichsan2895 thx for testing. I think that confirms what is to be expected. Awesome!

@ichsan2895 thx for testing. I think that confirms what is to be expected. Awesome!

Don't worry @MrNeRF

Now, I testing (again) with latest commit 139b521.

My result with scaffold-gs in commit 139b521

BTW, Can it save to ply and view it using supersplat or mkkellogg gaussian viewer?

BTW, Can it save to ply and view it using supersplat or mkkellogg gaussian viewer?

@ichsan2895
There is a way to do that.
I thought gsplat is creating ply files but (if I don't missed it), its not doing it. So I voted for saving checkpoints to be consistent.

One can also remove the view dependency from the mlps and it also works well but the measures are a bit worse. I think this is something that should be supported by Nerfstudio? How do you usually convert the checkpoints to ply files?

How do you usually convert the checkpoints to ply files?

For default densification strategies & MCMC strategies, this snippet of code in this thread works well:
Get .ply file after training?

AFAIK, since that code uses plyfile library which does not have compatible license with gsplat (GPLv3 vs Apache 2), The code is never merged yet. I dont know how to code it with Open3D (Apache 2.0).

I know how to do it. Is it like a big feature request? I can code it up in open3D. I just want to get this here done for a review.

I will be gone for a week and not coding. I can do it thereafter for both, scaffold and the rest.

I know how to do it. Is it like a big feature request? I can code it up in open3D. I just want to get this here done for a review.

It is not big feature request, but it is HUGE feature request 🥇

I will be gone for a week and not coding. I can do it thereafter for both, scaffold and the rest.

Happy holiday, cant wait for the next week

I think that's good for review now.

Amazing work @MrNeRF!
It doesn't seem like the current scaffold training is compatible with packed/sparse mode. Would be great if it had it integrated.

Is this draft support multiGPU? I test it with this commit e8d1207

I run this command:

SCENE_DIR="data/360_v2"
RESULT_DIR="results/benchmark_scaffold_2GPUs"
SCENE_LIST="garden bicycle stump bonsai counter kitchen room" # treehill flowers
RENDER_TRAJ_PATH="ellipse"

for SCENE in $SCENE_LIST;
do
    if [ "$SCENE" = "bonsai" ] || [ "$SCENE" = "counter" ] || [ "$SCENE" = "kitchen" ] || [ "$SCENE" = "room" ]; then
        DATA_FACTOR=2
    else
        DATA_FACTOR=4
    fi

    echo "Running $SCENE"

    # train without eval
    CUDA_VISIBLE_DEVICES=0,1 python3 examples/simple_trainer_scaffold.py --eval_steps 30000 --disable_viewer --data_factor $DATA_FACTOR \
        --render_traj_path $RENDER_TRAJ_PATH --steps_scaler 0.5 --packed \
        --data_dir data/360_v2/$SCENE/ \
        --result_dir $RESULT_DIR/$SCENE/
done

The error:

Traceback (most recent call last):
  File "/workspace/GSPLAT_SCAFFOLD3/gsplat/examples/simple_trainer_scaffold.py", line 1147, in <module>
    cli(main, cfg, verbose=True)
  File "/workspace/GSPLAT_SCAFFOLD3/gsplat/gsplat/distributed.py", line 344, in cli
    process_context.join()
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 163, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
    fn(i, *args)
  File "/workspace/GSPLAT_SCAFFOLD3/gsplat/gsplat/distributed.py", line 295, in _distributed_worker
    fn(local_rank, world_rank, world_size, args)
  File "/workspace/GSPLAT_SCAFFOLD3/gsplat/examples/simple_trainer_scaffold.py", line 1103, in main
    runner = Runner(local_rank, world_rank, world_size, cfg)
  File "/workspace/GSPLAT_SCAFFOLD3/gsplat/examples/simple_trainer_scaffold.py", line 336, in __init__
    self.splats, self.optimizers = create_splats_with_optimizers(
  File "/workspace/GSPLAT_SCAFFOLD3/gsplat/examples/simple_trainer_scaffold.py", line 190, in create_splats_with_optimizers
    features = torch.zeros((N, cfg.feat_dim))
NameError: name 'cfg' is not defined

@ichsan2895 No. I can't test it properly as I don't have a multi gpu setup.

Either I remove it to not make it crash or I implement it blindly.

@earth-bass Thx. I'll look into it.

My result with scaffold-gs in commit e8d1207

This commit slightly worse than previous commit. But, it is still good and not significant.

NEW SOTA! @MrNeRF 🔥

Absgrad increase significantly the metrics at the cost increasing of Num_GS and VRAM.

Garden, bicycle, and stump are OOM even I use cloud GPU (RTX A6000 48 GB).

Using commit e8d1207

Awesome! But it should not run oom. There is a vram intense part which is not optimally implemented. I can't fix it before next Sunday.

But if someone wants to jump in, feel free. I can point to the relevant code.

I enjoyed playing with different replacements of mlps and noise. That's why I didn't look much inro the vram consumption. But so far, I didn't manage to find anything that would give another quality boost.

However, absgrad helps but I didn't evaluate it more systematically. Thanks for running the benchmark. Did you also inspect it visually. In my experience absgrad tends to introduce additional floaters.

It amazes me that abs grad improves the results by 0.4 - 0.5 dB @ichsan2895. That's absolutely freaking crazy.

It means also that this implementation easily destroys the leading methods on the 3DGS leaderboard in mipnerf360. Some scenes are outperformed by 1dB PSNR.

It amazes me that abs grad improves the results by 0.4 - 0.5 dB @ichsan2895. That's absolutely freaking crazy.

It means also that this implementation easily destroys the leading methods on the 3DGS leaderboard in mipnerf360. Some scenes are outperformed by 1dB PSNR.

Yeah. Just fix the VRAM leak problem & feature for exporting PLY -> gets new SOTA -> create paper for next CVPR 😆

However, absgrad helps but I didn't evaluate it more systematically. Thanks for running the benchmark. Did you also inspect it visually. In my experience absgrad tends to introduce additional floaters.

So far, its good. It does not create extra unwanted floater in this kitchen scene. Needs to be inspect for another dataset.

Scaffold-GS

Scaffold-GS with Absgrad

Hey guys! It's super cool to see the the development here lead to new SOTA! However I would step back a little bit because the comparison here might be slightly unfair as this impl. produces more GSs, consumes more VRAM, and training time than the baselines. For almost all approaches of 3GDS, you could always get better performance by letting it converge to more GSs, with the price of spending more time and VRAM to train it. So I feel a super fair comparison should also consider training time & VRAM & number of GSs.

Btw, have you guys checked if this PR reimplements the performance of the original scaffold gs paper when following its original setup? I think when merging this pr, the default hyper parameters should be following the original paper, so that when user use it it should by default reproduce the paper's results. We then could support something more advanced like absgrad in the training script to get better performance, and log them in a README file or whatever.

There is currently a VRAM issue I need to resolve. It should be much less memory-hungry. But that won't impact the overrall structure of this pr.

The parameterization is different, as I spent some time fine-tuning it starting from what is given in the paper.

I can run some tests with the original parameters for comparison but thats already done on nerfbaselines. However, I noticed that the parameters and implementation from the paper do not align with key points highlighted in the paper. So what exactly should be compared?

Also, note that I made some steps in the densification algorithm optional, as I found they performed worse. Most notably, I replaced the pruning with MCMC relocation. I found that pruning triggered a much stronger densification, which did not translate into higher quality. The relocation seems to dampen this behavior in favour of less anchors and better quality. I also removed dropout, which is not justified in the paper and does not help. There are a few other differences as well. For instance, the schedulers used in gsplat are already different which will give also a different performance.

The voxelization, view-dependent MLPs, and anchor growth are implemented, which I believe are the main contributions of Scaffold-GS.

If I stay closer to the original Scaffold-GS (paper vs their code), the performance will be clearly worse. Absgrad can be off by default which I also did not activate in my tests.

Comparing against mcmc can only be fair with the same final budget as what Scaffold-GS uses. But again, for every anchor 10 neural gaussians are spawned. Most scenes will run oom with this mcmc budget on a 4090RTX.

Wow, one month later of this inactive thread.

If it not merged yet, please @MrNeRF don't remove this repo. It has big potential for giving the SOTA despite the slowness and heavy.

But I am curious how to export Gs-scaffold to PLY? Is it compatible with supersplat or mkkellogg gaussian-viewer?